feat(gateway): removal-refusal signal for the victim, and a node liveness gauge - #1102
Merged
Conversation
The removed-node lockout refuses with a 403 and a warn line, which
observes nothing: the audience who needs to notice is not reading
either node's logs.
That audience is the refused node itself. A mistakenly removed node is
exactly the machine whose monitoring is still attached -- and it
cannot learn of its removal from local state, because the refusals are
what keep its copy of the removal marker from ever replicating to it.
The signal has to be read off the door slamming. So the sync client
now keeps a non-success HTTP status as a typed error in the chain
(`HttpStatusError`) -- the sender must tell the lockout's 403 apart
from a peer that is down or broken, and string-matching a Display line
is not a contract -- and the sync network counts
`dstack_gateway_sync_rejected_as_removed_total` and logs the two ways
out (SetNodeUrl re-admission on a surviving gateway, or decommission)
whenever a peer answers 403. Nonzero has exactly one meaning: this
node has been removed and is still trying to sync.
The refusing side stays a warn line, deliberately: the counter that
sat there answered no question the victim-side one does not answer
better, and a correctly decommissioned box that is still powered shows
up in the new liveness gauge's staleness anyway.
Every gateway also exposes
`dstack_gateway_node_last_seen_timestamp_seconds{node=...}` -- the
latest time any gateway observed each known node, max across
observers, future-dated reports dropped. Until now a long-offline
gateway was visible only as an ack watermark that stopped advancing,
which takes a PromQL puzzle to alert on; this is `time() - metric >
threshold`. It also prices what an absent node costs the cluster:
while a member is unreachable, every tombstone written since its last
report is pinned.
Tests pin the two properties that make the victim-side counter
trustworthy -- only a 403 reads as a removal refusal (a peer that is
down, overloaded or broken must not claim this node was removed), and
the typed status survives a real TLS request's error chain -- plus the
exposition line of the liveness gauge.
kvinwang
force-pushed
the
feat/gateway-removal-refusal-metrics
branch
from
August 23, 2026 08:18
1ae0666 to
fde3338
Compare
kvinwang
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#1099 and #1100 have merged; this PR now targets
nextdirectly and is the last piece of the removed-node work.Problem
#1100's lockout refuses a removed node's envelopes with a 403 and a warn line — which observes nothing, because the audience who needs to notice is not reading either node's logs. And independently of removal, the cluster has no direct metric for a member that has simply gone quiet: a long-offline gateway is visible only as an ack watermark that stopped advancing, which takes a PromQL puzzle to alert on.
The refused node is the audience that matters
A mistakenly removed node is exactly the machine whose monitoring is still attached. And it cannot learn of its removal from local state: the removal marker replicates among the surviving members, and the refusals it drives are precisely what keep the victim's copy from ever arriving. The signal has to be read off the door slamming.
So the sync client now keeps a non-success HTTP status as a typed error in the chain (
HttpStatusError) — the sender must tell the lockout's 403 apart from a peer that is down or broken, and string-matching aDisplayline is not a contract — and the sync network, on a 403, countsdstack_gateway_sync_rejected_total. Sync endpoints use 403 for both removal lockouts and app-identity mismatches, so the counter and log report the rejection without claiming which condition caused it. Operators should correlate it with node membership and identity configuration.The refusing side deliberately stays a warn line, with no counter. It answered no question the victim-side counter does not answer better, and its one residual case — a correctly decommissioned box someone forgot to power off — shows up in the liveness gauge below regardless.
Node liveness, directly
dstack_gateway_node_last_seen_timestamp_seconds{node=...}— the latest time any gateway observed each known node (max across observers, future-dated reports dropped, same data the admin API already serves). Replicated, so aggregate withmax(); alert withtime() - metric > threshold.This is also the cost meter for the tombstone GC in #1099: while a member is unreachable, every tombstone written since its last report is pinned by its stale ack watermark, so "how long has node N been gone" is exactly "how much collection is this node holding up".
A node an operator removed leaves the series entirely (its records are deleted). Absence here, plus the 403 counter firing on the machine itself, can indicate the removed-by-mistake picture; an app-identity mismatch is the other possible cause.
Verification
cargo test -p dstack-gateway— 288 passed. Three new tests plus one exposition assertion:a_refusal_status_is_readable_from_the_error_chainonly_a_403_reads_as_rejecteda_rejection_is_counted_for_the_senders_own_monitoring